Load balancing system and method

ABSTRACT

Provided is a load balancing system and method capable of balancing the load of an apparatus employing both the internal mirror copying function and external connection function. With this load balancing system and method for balancing the load of a first apparatus equipped with a first function that copies data of one logical device to another logical device paired among the logical devices set in itself, and a second function that virtualizes an external logical device and provides it to a host system, the first and second apparatuses are controlled for periodically detecting the load of the first apparatus, and the load of a second apparatus provided separately from the first apparatus and equipped with the first and second functions, respectively, and migrating the logical device that is not paired with the other logical device among the logical devices set in the first apparatus to the second apparatus when the load of the first apparatus is greater than the load of the second apparatus.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application relates to and claims priority from Japanese PatentApplication No. 2005-315634, filed on Oct. 31, 2005, the entiredisclosure of which is incorporated herein by reference.

BACKGROUND

1. Field of the Invention

The present invention pertains to a load balancing system and methodwhich, for instance, are suitably employed in a storage system.

2. Description of the Related Art

Conventionally, as one application function installed in a storagedevice, there is a copy function for creating a mirror of a logicaldevice (this is hereinafter referred to as a “LDEV”) in the same storagedevice without going through a host system as the higher-level device(this is hereinafter referred to as an “internal mirror copyingfunction”).

Further, as another application function installed in a storage device,there is a function of virtualizing the LDEV set in an external storagedevice (this is hereinafter referred to as an “external LDEV”) andproviding it to the host system while making it seem that it is a LDEVin its own storage device (this is hereinafter referred to as an“internal LDEV”) (this hereinafter referred to as an “externalconnection function”).

In order to balance the load of a storage device that is not using theexternal connection function, it is standard to adopt the method ofmigrating (copying) data to a separate storage device via the hostsystem. Nevertheless, when the storage device is using the externalconnection function, since it will suffice to merely switch the mappingto the external LDEV to a LDEV in another storage device, the load ofthe storage device can be balanced relatively easily.

Incidentally, Japanese Patent Laid-Open Publication No. 2004-145855discloses technology for balancing the load of a storage device equippedwith a so-called remote copy function which overwrites data among aplurality of storage devices located in physically distant locationswithout the intervention of a host system.

SUMMARY

Nevertheless, as described later, in the case of a storage device thatuses both the internal mirror copying function and external connectionfunction, even if the switching processing for switching the mapping tothe external LDEV is performed, it is practically impossible to migratethe copy status before migration, and, as a result, there is a problemin that the load of a storage device that uses both the internal mirrorcopying function and external connection function cannot be balanced.

The present invention was devised in view of the foregoing points, andan object thereof is to provide a load balancing system and methodcapable of balancing the load of an apparatus using both the internalmirror copying function and external connection function.

The present invention for achieving the foregoing object is a loadbalancing system for balancing the load of a first apparatus equippedwith a first function that copies data of one logical device to anotherlogical device paired among the logical devices set in itself, and asecond function that virtualizes an external logical device and providesit to a host system, the load balancing system including: a secondapparatus equipped with the first and second functions; and a managementdevice that controls the first and second apparatuses for periodicallydetecting the load of the first apparatus and the load of the secondapparatus, respectively, and migrating the logical device that is notpaired with the other logical device among the logical devices set inthe first apparatus to the second apparatus when the load of the firstapparatus is greater than the load of the second apparatus.

As a result, with this load balancing system, when the load of the firstapparatus becomes greater than the load of the second apparatus, thelogical device set in the first apparatus is migrated to the secondapparatus, and the load of the first apparatus can thereby be balancedwith the second apparatus. Here, since the logical device to be migratedto the second apparatus is a logical device that is not paired withanother logical device, migration processing of such logical device canbe performed easily regardless of the current status of the respectivecopy pairs set in the first apparatus.

Further, in the present invention, the management device controls thefirst and second apparatuses for migrating the logical device that isnot paired with the other logical device among the logical devices setin the first apparatus to the second apparatus when the load of thefirst apparatus is greater than the load of the second apparatus, andthereafter migrating the logical device that is paired with the otherlogical device among the logical devices set in the first apparatus tothe second apparatus. Moreover, upon migrating a copy pair formed fromone logical device and another logical device paired in the firstapparatus to the second apparatus, the management device detects thecurrent status of the copy pair and notifies the second apparatus of thecontrol information according to the detection result; and the secondapparatus sets the status of the copy pair migrated from the firstapparatus to the second apparatus based on the control informationnotified from the management device.

As a result, with this load balancing system, the respective logicaldevices configuring a copy pair set in the first apparatus can bemigrated to the second apparatus while maintaining the copy pair statusat such time.

Further, the present invention also provides a load balancing method forbalancing the load of a first apparatus equipped with a first functionthat copies data of one logical device to another logical device pairedamong the logical devices set in itself, and a second function thatvirtualizes an external logical device and provides it to a host system,the load balancing method including: a first step of periodicallydetecting the load of the first apparatus, and the load of a secondapparatus provided separately from the first apparatus and equipped withthe first and second functions, respectively; and a second step ofcontrolling the first and second apparatuses for periodically detectingthe load of the first apparatus and the load of the second apparatus,respectively, and migrating the logical device that is not paired withthe other logical device among the logical devices set in the firstapparatus to the second apparatus when the load of the first apparatusis greater than the load of the second apparatus.

As a result, with this load balancing method, when the load of the firstapparatus becomes greater than the load of the second apparatus, thelogical device set in the first apparatus is migrated to the secondapparatus, and the load of the first apparatus can thereby be balancedwith the second apparatus. Here, since the logical device to be migratedto the second apparatus is a logical device that is not paired withanother logical device, migration processing of such logical device canbe performed easily regardless of the current status of the respectivecopy pairs set in the first apparatus.

Further, in the present invention, at the second step, the first andsecond apparatuses are controlled for migrating the logical device thatis not paired with the other logical device among the logical devicesset in the first apparatus to the second apparatus when the load of thefirst apparatus is greater than the load of the second apparatus, andthereafter migrating the logical device that is paired with the otherlogical device among the logical devices set in the first apparatus tothe second apparatus. Moreover, at the second step, upon migrating acopy pair formed from one logical device and another logical devicepaired in the first apparatus to the second apparatus, the currentstatus of the copy pair is detected and the control informationaccording to the detection result is notified to the second apparatus;and wherein second apparatus sets the status of the copy pair migratedfrom the first apparatus to the second apparatus based on the controlinformation notified from the management device.

As a result, with this load balancing method, the respective logicaldevices configuring a copy pair set in the first apparatus can bemigrated to the second apparatus while maintaining the copy pair statusat such time.

According to the present invention, the load of an apparatus using botha first function that copies data of one logical device to anotherlogical device paired among the logical devices set in itself, and asecond function that virtualizes an external logical device and providesit to a host system can be easily balanced regardless of the currentstatus of the respective copy pairs set in the first apparatus.

DESCRIPTION OF DRAWINGS

FIG. 1 is a conceptual diagram for explaining the internal mirrorcopying function in a storage device;

FIG. 2 is a conceptual diagram for explaining the internal mirrorcopying function in a storage device;

FIG. 3 is a conceptual diagram for explaining the internal mirrorcopying function in a storage device;

FIG. 4 is a conceptual diagram for explaining the internal mirrorcopying function in a storage device;

FIG. 5 is a transition diagram for explaining the internal mirrorcopying function in a storage device;

FIG. 6 is a flowchart for explaining the internal mirror copyingfunction in a storage device;

FIG. 7 is a flowchart for explaining the internal mirror copyingfunction in a storage device;

FIG. 8 is a conceptual diagram for explaining the external connectionfunction in a storage device;

FIG. 9 is a conceptual diagram for explaining the interaction of theinternal mirror copying function and external connection function;

FIG. 10 is a conceptual diagram for explaining the migration processingof LDEV via a host system;

FIG. 11 is a conceptual diagram for explaining the load balancingprocessing when using the external connection function;

FIG. 12 is a conceptual diagram for explaining the problems of loadbalancing processing when using the internal mirror copying function andexternal connection function;

FIG. 13 is a block diagram showing the configuration of the storagesystem according to the first to third embodiments;

FIG. 14 is a block diagram showing the configuration of the existedstorage system, additional storage device and external storage device;

FIG. 15 is a conceptual diagram showing the connection relationship ofthe load balancing software, LDEV management table, pair managementtable and differential bit information;

FIG. 16(A) is a conceptual diagram for explaining the LDEV managementtable, and FIG. 16(B) is a chart for explaining the LDEV managementtable;

FIG. 17 is a conceptual diagram for explaining the pair managementtable;

FIG. 18 is a conceptual diagram for explaining the differential bitinformation;

FIG. 19 is a chart for explaining the various commands relating to loadbalancing processing

FIG. 20 is a chart for explaining the various commands relating to loadbalancing processing;

FIG. 21(A)-(B) are conceptual diagrams for explaining the“createMapping” command;

FIG. 22(A)-(B) are conceptual diagrams for explaining the“deleteMapping” command;

FIG. 23 is a flowchart for explaining the “createpair” command;

FIG. 24 is a flowchart showing the first load balancing processingsequence;

FIG. 25 is a flowchart showing the first LDEV migration processingsequence;

FIG. 26 is a flowchart showing the table acquisition processingsequence;

FIG. 27 is a flowchart showing the load balance check processingsequence;

FIG. 28 is a flowchart showing the first LDEV search processingsequence;

FIG. 29 is a flowchart showing the path switching processing sequence;

FIG. 30 is a flowchart showing the second LDEV migration processing;

FIG. 31 is a flowchart showing the first copy pair search processingsequence;

FIG. 32 is a flowchart showing the third LDEV migration processingsequence;

FIG. 33 is a flowchart showing the fourth LDEV migration processingsequence;

FIG. 34 is a flowchart showing the fifth LDEV migration processingsequence;

FIG. 35 is a flowchart showing the second copy pair search processingsequence;

FIG. 36 is a flowchart showing the second load balancing processingsequence;

FIG. 37 is a flowchart showing the sixth LDEV migration processingsequence;

FIG. 38 is a flowchart showing the sixth LDEV migration processingsequence;

FIG. 39 is a flowchart showing the third load balancing processingsequence;

FIG. 40 is a block diagram showing the configuration of the storagesystem according to the fourth embodiment; and

FIG. 41 is a block diagram showing the configuration of the installedfibre channel switch and additional fibre channel switch.

DETAILED DESCRIPTION

An embodiment of the present invention is now described in detail withreference to the drawings.

(1) Application Function in Storage Device

(1-1) Internal Mirror Copying Function

With a storage device 2 equipped with an internal mirror copyingfunction, upon executing the internal mirror copying function, as shownin FIG. 1, by copying in advance the data stored in a LDEV (this ishereinafter referred to as a “primary volume”) 3 to become the copysource to a LDEV (this is hereinafter referred to as a “secondaryvolume”) 4 to become the copy destination among the two LDEVs 3, 4 setas a copy pair (this is hereinafter referred to as a “pair setting” or“paired”), contents of the primary volume 3 and secondary volume 4 aremade to be the same.

And, when a host system 1 thereafter issues a data write request to theprimary volume 3, the storage device 2 writes the data in a designatedaddress of the primary volume 3 according to such write request, andfurther writes the same data in a corresponding address of the secondaryvolume 4 synchronously or asynchronously.

Further, when the host system 1 thereafter issues a data write requestto the primary volume 3 in a state where the mirror of the primaryvolume 3 and secondary volume 4 is cancelled and a snapshot is created,the storage device 2 writes the data to be written in the primary volume3, and concurrently stores the location thereof.

As a means for realizing the above, the storage device 2 retains in aninternal memory 5 a bit string having the same number of bits as thenumber of blocks, which is the access unit of data in the primary volume3, as differential bit information 6, and manages the difference betweenthe primary volume 3 and secondary volume 4 by setting “0” as the valueof the corresponding bit of the differential bit information 6 forblocks in which the contents of the primary volume 3 and secondaryvolume 4 are the same, and setting “1” as the value of the correspondingbit for blocks in which the contents of the primary volume 3 andsecondary volume 4 are different.

Meanwhile, in this internal mirror copying function, as the status ofthe primary volume 3 and secondary volume 4 (this is hereinafterreferred to as a “pair status”), there are three statuses; namely,“pair”, “copy” and “split”. And, with respect to the respective pairedprimary volumes 3 and secondary volumes 4, the storage device 2 equippedwith the internal mirror copying function manages the LDEV number of theprimary volume 3 and secondary volume 4 and the current pair status ofthe primary volume 3 and secondary volume 4 with a pair management table7 stored in the internal memory 5.

Here, the pair status of “pair”, as shown in FIG. 2, indicates a statuswhere the data copy from the primary volume 3 to the secondary volume 4is complete and the primary volume 3 and secondary volume 4 aremirrored. During this pair status, when data is written in the primaryvolume 3, the same data is simultaneously written in the correspondingblock of the secondary volume 4. Further, during this pair status, sincethe contents of the primary volume 3 and secondary volume 4 are thesame, all bits of the differential bit information 6 are constantly “0”.

The pair status of “copy”, as shown in FIG. 3, is a status there thedata copy from the primary volume 3 to the secondary volume 4 is midway,and the primary volume 3 and secondary volume 4 are not yet mirrored.During this pair status, when data is written in the primary volume 3,the same data is simultaneously written in the corresponding location ofthe secondary volume 4, and, in parallel therewith, data copy of theblock in which the differential bit information 6 is set to “1” isperformed between the primary volume 3 and secondary volume 4.

Further, the pair status of “split”, as shown in FIG. 4, is a statuswhere data is not written in the secondary volume 4, and this secondaryvolume 4 creates a snapshot at a certain point in time. During this pairstatus, even when data is written in the primary volume 3, data is notwritten in the secondary volume 4, and “1” is set to the correspondingbit of the differential bit information 6.

FIG. 5 shows the transition of the status between the respective pairstatuses of “pair”, “copy” and “split”, and the pair status with no pairsetting (“no pair”). As evident from FIG. 5, when two LDEVs are pairedfrom the pair status of “no pair”, the pair status of such pair changesto “copy”, and, when the data copy from the primary volume 3 to thesecondary volume 4 is completed thereafter, the pair status changes to“pair”.

And, when a pair split request is issued to the storage device 2 fromthe pair status of “pair”, the pair status of the primary volume 3 andsecondary volume 4 becomes “split”, and, by making a resynchronizationrequest from the pair status of “split”, the pair status can be changedonce again to “copy”. Incidentally, in each of the cases of “pair”,“copy” and “split”, the pair status can be changed to “no pair” byerasing the pair.

FIG. 6 is a flowchart representing the processing to be performed by thestorage device 2 in relation to the data write processing to the primaryvolume 3 upon executing the internal mirror copying function.

When the host system 1 issues a data write request to the primary volume3, the storage device 2 foremost writes the data to be written providedfrom the host system 1 together with the write request in a designatedblock of the primary volume 3 (SP1).

Further, the storage device 2 thereafter refers to the pair managementtable 7 (FIG. 1), and sets “1” to the corresponding bit of thedifferential bit information 7 (FIG. 1) (SP3) when the pair status ofthe pair number given to the copy pair formed from the primary volume 3and the corresponding secondary volume 4 is “split” (SP2: YES).

Contrarily, when the pair status of the pair of the primary volume 3 andthe secondary volume 4 is not “split” (SP2: NO), the storage device 2sets “0” to the corresponding bit of the differential bit information 7(SP4), and writes the same data as the data written in the primaryvolume 3 in the corresponding block of the secondary volume 4 (SP5).

Meanwhile, FIG. 7 is a flowchart representing the processing to beperformed by the storage device 2 in relation to the data writeprocessing (this is hereinafter referred to as “data copy processing”)to the secondary volume 4. The storage device 2 executes, in prescribedintervals, the data copy processing to the secondary volume 4asynchronously with the data write processing to the primary volume 3according to this flowchart regarding the pairs formed by the primaryvolume 3 and secondary volume 4 having a pair status of “pair” or“copy”.

In other words, when the storage device 2 starts the data copyprocessing, it foremost selects one pair of the primary volume 3 andsecondary volume 4 registered in the pair management table 5 (FIG. 1),and determines whether or not the pair status of this copy pair is“split” based on the pair management table 7 (SP10).

When the pair status of this pair is “split” (SP10: YES), the storagedevice 2 ends this data copy processing, and, when the pair status ofthis pair is not “split” (SP10: NO), it searches a bit having a value of“1” from the corresponding differential bit information (SP11).

When the storage device 2 is not able to detect a bit of “1” in thissearch (SP12: NO), it ends this data copy processing, and, when thestorage device 2 is able to detect a bit of “1” (SP12: YES), it readsthe block data corresponding to this bit in the primary volume 3 (SP13),and writes this in the corresponding block of the secondary volume 4(SP14).

Thereafter, the storage device 2 changes the value of the bit determinedto be “1” at step SP12 in the corresponding differential bit informationto “0”, and then ends this data copy processing thereafter (SP16).

(1-2) External Connection Function

Meanwhile, the storage device 2 equipped with the external connectionfunction, upon executing this external connection function, maps a LDEV12 in an external storage device 10 connected to an external connectionport 2A to a virtual LDEV 11 set in itself. And, when the host system 1accesses the virtual LDEV 11, the storage device 2 issues acorresponding data I/O request to the external storage device 10 via theexternal connection port 2A. As a result, the corresponding data isinput and output to and from the corresponding block in the LDEV 12 ofthe external storage device 10. Accordingly, the access from the hostsystem 1 to the virtual LDEV 11 is actually made to the LDEV 12 of theexternal storage device 10.

Further, it is also possible to coordinate this kind of externalconnection function and the foregoing internal mirror copying function.Here, as shown in FIG. 9, the external connection function of thestorage device 2 is used to respectively map the first and second LDEVs12A, 12B of the external storage device 10 to the first and secondvirtual LDEVs 11A, 11B of the storage device, and the internal mirrorcopying function is used to copy data between these first and secondvirtual LDEVs 11A, 11B.

However, since the first and second virtual volumes 12A, 12B have nomateriality, access from the host system 1 to the first virtual LDEV 11Ais made to the first LDEV 12A via the first virtual LDEV 11A, and thedata copy from the first virtual LDEV 11A to the second virtual LDEV 12Ais executed as the data copy from the first LDEV 12A to the second LDEV12B.

(1-3) Load Balancing Processing

Meanwhile, when a LDEV set in a certain first storage device is accessedfrequently and the load of the first storage device is significant,considered may be a case of balancing the load of the first storagedevice by migrating the data stored in the LDEV of such first storagedevice to a LDEV of another second storage device.

In the foregoing case, as shown in FIG. 10, data migration in a casewhen the first storage device 20 is not using the external connectionfunction can be performed by reading data from a target LDEV 22 in thefirst storage device 11 and writing this in a target LDEV 24 of thesecond storage device 23 via the host system 1 based on the control ofdata migration software 20 loaded in the host system 1. Nevertheless,with this method, there is a problem in that the host system 1, as wellas the first and second storage devices 21, 23, will be burdened sincethis involves the actual migration of data.

Contrarily, as shown in FIG. 11, when the first storage device 30 isusing the external connection function; that is, with the data migrationin a case where a LDEV (this kind of LDEV is hereinafter referred to asan “external LDEV”) 33 set in the second storage device 32 connected tothe external connection port 31 of the first storage device 30 is mappedto the first virtual LDEV 34 set in the first storage device 30, theload of the first storage device 30 can be balanced with the thirdstorage device 35 merely by re-mapping the external LDEV 33 to theexternal LDEV 33 and the second virtual LDEV 36 set in the third storagedevice 35. Here, by assigning the virtual LDEV 36 of the third storagedevice 35 from a port 37 in the third storage device 35 to a port 38 ofthe host system 1 with a prescribed LUN number, the host system 1 willbe able to access the external LDEV 33 via the second virtual volume 36.

And, since this method does not involve the actual migration of data,there is an advantage in that the load of the first storage device 30can be balanced without burdening the host system 1 as well as the firstand second storage devices 30, 33 with the load caused by the datamigration.

Nevertheless, with the storage device 2 using both the externalconnection function and internal mirror copying function as describedwith reference to FIG. 9, the pair management table 7 and thedifferential bit information 6 of each copy pair exist in the firststorage device 2. Thus, for instance, as shown in FIG. 12, in a casewhere the first and second external LDEVs 33A, 33B set in the secondstorage device 32, which is an external storage device, are respectivelymapped to the first and second virtual LDEVs 34A, 34B of the firststorage device 30, a third storage device 35 will not be able to succeedthe pair management table 7 and differential bit information 11B bymerely mapping these first and second external LDEVs 33A, 33B to thethird or fourth virtual LDEV 36A, 36B of the third storage device 35.

In other words, with the system configured as illustrated in FIG. 12, bymerely performing processing for switching the mapping to the externalLDEVs 33A, 33B in the second storage device 32 to the mapping from thefirst and second virtual LDEVs 34A, 34B in the first storage device 30to the first and second virtual LDEVs 36A, 36B in the third storagedevice 35, there is a problem in that the volume cannot be migrated fromthe first storage device 30 to the third storage device 35, and, as aresult, the load of the first storage device 30 cannot be balanced.

In particular, when the first and second virtual volumes 34A, 34B in thefirst storage device 30 have a pair status of “pair”, the load of thefirst storage device 30 cannot be balanced with the third storage device35 while maintaining the mirrored status.

The storage system according to the first to fourth embodimentsdescribed below, as shown in FIG. 13, has one characteristic in that itis able to balance the load of an existing storage device (this ishereinafter referred to as an “existed storage system”) executing theinternal mirror copy with the internal mirror copying function betweentwo virtual LDEVs respectively mapped to the external LDEV with theexternal connection function with another storage device (this ishereinafter referred to as an “additional storage device”) whilemaintaining the current status of the copy pair (data consistencybetween the pair status and primary/secondary volume).

The storage system according to the first to fourth embodiments equippedwith this kind of function (this is hereinafter referred to as a “loadbalancing function”) is now explained.

(2) First Embodiment

(2-1) Configuration of Storage System According to First Embodiment

In FIG. 13, reference numeral 40 represents the overall storage systemaccording to the first embodiment. This storage system 40 is configuredby a host system 41, an existed storage system 42, an additional storagedevice 43 and a management server 44 being connected via a first network45; the existed storage system 42 and additional storage device 43 beingconnected to an external storage device 47 via a second network 46; andthe host system 41, existed storage system 42, additional storage device43, management server 44 and external storage device 47 being connectedvia a LAN (Local Area Network) 48.

The host system 41 as the higher-level device is a computer devicehaving information processing resources such as a CPU (CentralProcessing Unit) and memory, and, for instance, is configured from apersonal computer, workstation, mainframe or the like. The host system41 has an information input device (not shown) such as a keyboard,switch or pointing device, and an information output device (not shown)such as a monitor device or a speaker. Further, the host system 41 isalso provided with an HBA (Host Bus Adapter) 50 that functions as aninterface for accessing the existed storage system 42 or the like viathe first network 45, and an NIC (Network Interface Card) 51 forcommunicating with the management server 44 or the like via the LAN 48.

The existed storage system 42 has an external connection function whichvirtualizes a LDEV (this is hereinafter referred to as an “externalLDEV”) 52 provided to the host system 41 by the external storage device47 and provides this to the virtual LDEV 53, and an internal mirrorcopying function which performs the copy processing between the LDEVs(virtual LDEV 53 and internal LDEV 62 described later) set in itself,and, as shown in FIG. 14, is configured by having a plurality ofphysical storage devices 60 and a control unit 61.

Among the above, as the physical storage devices 60, for example, anexpensive disk such as a SCSI (Small Computer System Interface) disk oran inexpensive disk such as a SATA (Serial AT Attachment) disk or anoptical disk may be used.

These physical storage disks 60 are operated with the control unit 61according to the RAID system. One or more LDEVs (these are hereinafterreferred to as “internal LDEVs”) 62 (FIG. 13) are set on a physicalstorage area provided by one or more physical storage disks 60. And datais stored in block (this is hereinafter referred to as a “logicalblock”) units of a prescribed size in these internal LDEVs 62 andvirtual LDEVs 53.

Each internal LDEV 62 and each virtual LDEV 53 is given a uniqueidentifier (this is hereinafter referred to as a “LDEV number”). In thecase of this embodiment, the input and output of data is conducted bydesignating an address which is a combination of this LDEV number and aunique number given to each block (LBA: Logical Block Address).

Meanwhile, the control unit 61 is configured by including a plurality ofchannel adapters 70, a connection 71, a shared memory 72, a cache memory73, a plurality of disk adapters 74 and a management terminal 75.

Each channel adapter 70 is configured as a microcomputer system having amicroprocessor, memory and communication interface, and interprets thevarious commands transmitted from the host system 41 (FIG. 13) via thefirst network 45 and executes the corresponding processing.

Further, each channel adapter 70, as shown in FIG. 13, has a normal port42A for connecting with the first network 45, and an external connectionport 42B having an initiator function and capable of issuing a SCSIcommand. The port 42A and external connection port 42B are assigned aport address such as an IP (Internet Protocol) address or WWN (WorldWide Address) for identifying the respective ports, and each channeladapter 70 is thereby able to independently behave as a NAS (NetworkAttached Storage).

The connection 71 is connected to each channel adapter 70, shared memory72, cache memory 73 and each disk adapter 74. The transfer of data andcommands among the channel adapters 70, shared memory 72, cache memory73 and disk adapters 74 is conducted via this connection 71. Theconnection 71, for instance, is configured from a switch such as anultra-speed crossbar switch or bus for transferring data via high-speedswitching.

The shared memory 72 and cache memory 73 are memories to be shared bythe channel adapters 70 and disk adapters 74. The shared memory 72 ismainly used for storing system configuration information relating to theconfiguration of the overall storage device and commands. The LDEVmanagement table 85 described later, pair management table 7 anddifferential bit information 6 of each copy pair are also stored in theshared memory 72. Further, the cache memory 73 is mainly used fortemporarily storing data to be input and output to and from the physicalstorage devices 60.

Each disk adapter 74 is configured as a microcomputer system having amicroprocessor, memory and the like, and functions as an interface forcontrolling the protocol upon communicating with the physical storagedevices 60. These disk adapters 74, for example, are connected to thecorresponding physical storage devices 60 via a fibre channel cable, andthe transfer of data with such physical storage devices 60 is conductedaccording to a fibre channel protocol.

The management terminal 75 is a terminal device for controlling theoverall operation of the storage device and, for instance, is configuredfrom a node-type personal computer. The management terminal 75 isconnected to each channel adapter 70 and each disk adapter 74 via a LANnot shown. The management terminal 75 monitors the occurrence of failurein the storage device, and indicates the occurrence of a failure on itsdisplay and performs the close processing of the corresponding physicalstorage devices according to the operator's operations. The operator isable to define the system configuration information with the managementterminal 75, and store this defined system configuration information inthe shared memory 72 via the channel adapters 70 or the disk adapters 74and connection 71.

Similarly, the additional storage device 43 has an external connectionfunction for virtualizing the LDEV 52 provided to the host system 41 bythe external storage device 47 and providing this to the virtual LDEV 76(FIG. 13), and an internal mirror copying function for performing copyprocessing between the virtual LDEV 76 and internal LDEV 77 set initself (FIG. 13), and, as shown in FIG. 13, is configured by having aplurality of physical storage devices 60 and a control unit 61.

This additional storage device 43 has the same configuration as theexisted storage system 42 described above with reference to FIG. 14, andis connected to the first network 4 via the port 43A and connected tothe second network 46 via the external connection port 43B.

The management server 44 is configured by comprising a CPU 80 forexecuting the various operations, a ROM (Read Only Memory) not shownstoring various control programs, a RAM (Random Access Memory) not shownas the work memory of the CPU 80, a physical storage devices 81configured from a hard disk or the like, an NIC 82 for connecting to theLAN, and an HBA 83 for accessing the host system 41 or existed storagesystem 42 or additional storage device 43 via the first network 45.

The physical storage devices 81 store load balancing software 84prescribing the processing for controlling the load balancing of theexisted storage system 42 in relation to the data I/O processing anddata copy processing, and the CPU 80 checks the load balance of theexisted storage system 42 and additional storage device 43 according tothis load balancing software 84 as described later, and controls theexisted storage system 42, additional storage device 43 and externalstorage device 47 when the load of the existed storage system 42 ishigher, and balances the load of the existed storage system 42 with theadditional storage device 43.

The first and second networks 45, 46, for instance, are configured froma SAN (Storage Area Network), LAN, Internet, public line or dedicatedline. Communication via such first or second network 45, 46 is conductedaccording to a fibre channel protocol when the first or second network45, 46 is a SAN, and conducted according to a TCP/IP (TransmissionControl Protocol/Internet Protocol) when the first or second network 45,46 is a LAN. Incidentally, in this embodiment, although the first andsecond networks 45, 46 are configured separately for facilitating theunderstanding of the connection relationship among the existed storagesystem 42, additional storage device 43 and external storage device 47,these may also be configured from a single network.

Further, the external storage device 47 is configured the same as theexisted storage system 42 described above with reference to FIG. 14excluding the point that it does not have the external connectionfunction and internal mirror copying function, and is connected to theexternal connection port 42B of the existed storage system 42 and theexternal connection port 43B of the additional storage device 43 via thesecond network 46.

(2-2) Load Balancing Processing Function in Storage System

(2-2-1) Configuration of Various Tables and Differential Bit Information

Next, before explaining the load balancing processing function in thestorage system 40 of this embodiment, the configuration of the varioustables and differential bit information to be used upon executing theload balancing processing function is explained.

As shown in FIG. 15, the respective shared memories 72 of the existedstorage system 42 and additional storage device 43 retain the LDEVmanagement table 85, pair management table 7 and differential bitinformation 6. And, the CPU 80 of the management server 44, upon loadbalancing processing, acquires the data of the LDEV management table 85,pair management table 7 and differential bit information 6 from theexisted storage system 42 and additional storage device 43 via the NIC82 and LAN 48.

Here, the LDEV management table 85 is a table for retaining informationrelating to all internal LDEVs 62, 77 and virtual LDEVs 53, 76 existingon the existed storage system 42 or additional storage device 43, and,as shown in FIG. 16(A), is configured from an internal (virtual) LDEVfield 90 and an external LDEV field 91.

Among the above, the internal (virtual) LDEV field 90 is configured froman LDEV number column 92, a capacity column 93, a LUN column 94 and anexternal connection port address column 95. And, the LDEV number column92 stores the serial number respectively given to each internal LDEV 62,77 or virtual LDEV 53, 76, and the capacity column 93 stores thecapacity of the corresponding internal LDEVs 62, 77 or virtual LDEVs 53,76. Moreover, the LUN column 94 stores the LUN of the correspondinginternal LDEVs 62, 77 or virtual LDEVs 53, 76, and the externalconnection port address column 95 stores the port address of theexternal connection ports 42A, 43A (FIG. 13) in the existed storagesystem 42 or additional storage device 43.

Further, the external LDEV field 91 is configured from a port addresscolumn 96 and a LUN column 97. And, when the corresponding LDEV on theexisted storage system 42 or additional storage device 43 is a virtualLDEV 53, 76, the LUN of the external LDEV 52 (FIG. 13) in an externalstorage device 47 (FIG. 13) mapped to the virtual LDEV 53, 76 is storedin the LUN column 97, and, for instance, the port address of the port47A (FIG. 13) in the external storage device 47 connected to theexternal LDEV 52 is stored in the port address column 96.

Accordingly, as shown in FIG. 16(B), with respect to an entry storingLUN in both the LUN column 94 of the internal (virtual) LDEV field 90and the LUN column 97 of the external LDEV field 91 in the LDEVmanagement table 85, the actual data exists in the external LDEV 52(FIG. 13), and is accessible from the host system 41 (FIG. 13). Withrespect to an entry storing LUN in the LUN column 94 of the internalLDEV field 90, but not storing LUN in the LUN column 97 of the externalLDEV field 91, the actual data exists in the internal LDEVs 62, 77 ofthe existed storage system 42 or additional storage device 43, and isaccessible from the host system 41. Incidentally, an entry not storingLUN in both the LUN column 94 of the internal (virtual) LDEV field 90and the LUN column 97 of the external LDEV field 91 is an unused LDEVset on the existed storage system 42 or additional storage device 43,and is inaccessible from the host system 41.

Meanwhile, the pair management table 7 is a table for retainingconfiguration information of the copy pair paired for the copyprocessing in the internal mirror using the internal mirror copyingfunction, and, as shown in FIG. 17, is configured from a pair numbercolumn 100, a primary volume LDEV number column 101, a secondary volumeLDEV number column 102 and a pair status column 103.

Here, the pair number column 100 stores a pair number which is a uniquenumber given to the corresponding copy pair, and the primary volume LDEVnumber column 101 stores a LDEV number of the LDEV forming a primaryvolume of such copy pair. Further, the secondary volume LDEV numbercolumn 102 stores a LDEV number of the LDEV forming a secondary volumeof such copy pair, and the pair status column 103 stores the currentpair status (“pair”, “copy” or “split”) of the copy pair.

Meanwhile, the differential bit information 6, as shown in FIG. 18, is abit string configured from the same number of bits as the number of LDEVblocks forming the primary volume and secondary volume of thecorresponding copy pair, and “1” is set to a bit corresponding to ablock in which the data of the primary volume and secondary volume aredifferent, and “0” is set to a bit corresponding to a block in which thedata of the primary volume and secondary volume are the same. Thisdifference bit row 6 exists in the same number as the number of copypairs by respectively corresponding to the copy pairs registered in thepair management table 7.

(2-2-2) Various Commands Relating to Load Balancing Processing

Next, the command used by the CPU 80 (FIG. 13) of the management server44 upon executing the load balancing processing is explained.

In the foregoing storage system 40, the CPU 80 of the management server44, upon executing the load balancing function, causes the existedstorage system 42 or additional storage device 43 or external storagedevice 47 to execute required processing by transmitting a correspondingcommand among the various commands shown in FIG. 19 and FIG. 20 to theexisted storage system 42 or additional storage device 43 or externalstorage device 47 via the LAN 48 (FIG. 13) based on the load balancingsoftware 84 (FIG. 13).

FIG. 19 and FIG. 20 show an example of the various commands transmittedfrom the management server 44 to the existed storage system 42 oradditional storage device 43 or external storage device 47 upon theexecution of the load balancing function described above.

For example, a “createMapping” command is used for instructing theexisted storage system 42 or additional storage device 43 to designatethe LDEV number, LDEV capacity, port address of the storage device to bemapped and LUN of the external LDEV 52 to be mapped, create virtualLDEVs 53, 76 having the same capacity as the LDEV of the LDEV number,and map the external LDEV 52 of the LUN assigned under the port of theport address connected to its external connection ports 42B, 43B to thecreated virtual LDEVs 53, 76. As a result of this processing, a singleexternal LDEV 52 and a single virtual volume 53, 76 will be associated.

Specifically, the CPU 80 of the management server 44, as shown in FIG.21(A), creates a virtual LDEV 53 in which the LDEV number is “A” and thecapacity is “100 (GB)” in the existed storage system 42, and transmits acommand of “createmapping A 100 12.23.34.45.56.67.78.89 n” when suchvirtual LDEV 53 is to be mapped to the external LDEV 52 in which the LUNin the external storage device 47 is “n” and the network address of theport 47A connected to the external connection port 42B of the existedstorage system 42 is “12.23.34.45.56.67.78.89”.

As a result, the existed storage system 42 given this command, as shownin FIG. 21(B), creates the virtual LDEV 53 of the designated LDEV numberand capacity by registering the designated LDEV number and capacity, andthe network address of the external connection port 42B in the internal(virtual) LDEV field 90 of the LDEV management table 85 according tothis command. Further, the existed storage system 42 maps the designatedexternal LDEV 52 to the virtual LDEV 53 by storing the designatednetwork address and designated LUN in the external LDEV field 91 of theentry in the LDEV management table 85. The existed storage system 42 atsuch time transmits a command of “True” when the processing is a successand a command of “Failure” when the processing is a failure to themanagement server 44 via the LAN 48.

Meanwhile, a “deleteMapping” command is used for instructing the existedstorage system 42 or additional storage device 43 to delete theassociation of the external LDEV 52 and virtual LDEVs 53, 76, and deletethe entry of the corresponding LUN from the LDEV management table 85. Asa result of this processing, the host system 41 will not be able toaccess the virtual LDEVs 53, 76.

For example, the CPU 80 of the management server 44, as shown in FIG.22(A), transmits a command of “deleteMapping A” to the existed storagesystem 42 when it wishes to delete the virtual LDEV 53 in which the LDEVnumber in the existed storage system 42 is “A”.

As a result, the existed storage system 42 given this command delete themapping of the external storage device 47 mapped to the externalconnection port 42B according to this command, deletes the virtual LDEV53 in which the LDEV number is “A”, and, as shown in FIG. 22(B), deletesthe entry of this virtual LDEV 53 from this LDEV management table 85.Here, this existed storage system 42 transmits a command of “True” whenthe processing is a success and a command of “Failure” when theprocessing is a failure to the management server 44 via the LAN 48.

Meanwhile, a “createpair” command is used for designating the LDEVnumber of the copy source, LDEV number of the copy destination and thepair status (“copy”, “pair” or “split”) of the copy pair to be created,and used for creating a copy pair in the existed storage system 42 oradditional storage device 43. Here, as the argument corresponding to therespective pair statuses of “copy”, “pair” or “split”, “-inti”, “-pair”or “-split” is used, respectively.

For example, the CPU 80 of the management server 44 transmits a commandof “createpair A B -pair” to the existed storage system 42 or additionalstorage device 43 when it wishes to create a copy pair in which the pairstatus is “pair” with the virtual LDEVs 53, 76 or internal LDEVs 62, 77having “A” as the LDEV number in the existed storage system 42 oradditional storage device 43 being the copy source (primary volume), andthe virtual LDEVs 53, 76 or internal LDEVs 62, 77 having “B” as the LDEVnumber being the copy destination (secondary volume).

FIG. 23 is a flowchart showing the processing to be performed by thechannel adapter 70 in the existed storage system 42 or additionalstorage device 43 that received this “createpair” command. The channeladapter 70 creates a designated copy pair according to this flowchart.

In other words, when the channel adapter 70 receives the “createpair”command, it foremost determines whether or not the argument regardingthe pair status contained in this “createpair” command is “-init”(SP20), and proceeds to step SP23 when it obtains a negative result(SP20: NO).

Meanwhile, when the channel adapter 70 obtains a positive result in thedetermination at step SP20 (SP20: YES), it newly registers thedesignated copy pair in the pair management table 7. Specifically, thepair number of this copy pair, LDEV number of the virtual LDEVs 53, 76or internal LDEVs 62, 77 to become the primary volume (LDEV number ofthe copy source LDEV designated in the command), LDEV number of thevirtual LDEVs 53, 76 or internal LDEVs 62, 77 to become the secondaryvolume (LDEV number of the copy destination LDEV designated in thecommand) and the pair status (“copy”) are respectively added to the pairmanagement table 7 (SP21). And, the channel adapter 70 thereaftercreates differential bit information 6 associated with this copy pair,and sets “1” to all bits in this differential bit information 6 (SP22).

Next, the channel adapter 70 determines whether or not the argumentregarding the pair status contained in the “createpair” command is“-pair” (SP23), and proceeds to step SP26 when it obtains a negativeresult (SP23: NO).

Meanwhile, when the channel adapter 70 obtains a positive result in thedetermination at step SP23 (SP23: YES), it newly registers thedesignated copy pair in the pair management table 7. Specifically, as inthe case at step SP21, the pair number of this copy pair, LDEV number ofthe virtual LDEVs 53, 76 or internal LDEVs 62, 77 to become the primaryvolume, LDEV number of the virtual LDEVs 53, 76 or internal LDEVs 62, 77to become the secondary volume and the pair status (“pair”) arerespectively added to the pair management table 7 (SP24). And, thechannel adapter 70 thereafter creates differential bit information 6associated with this copy pair, and sets “0” to all bits in thisdifferential bit information 6 (SP25).

Next, the channel adapter 70 determines whether or not the argumentregarding the pair status contained in the “createpair” command is“-split” (SP26), and ends the series of processing steps when it obtainsa negative result (SP26: NO).

Meanwhile, when the channel adapter 70 obtains a positive result in thedetermination at step SP26 (SP26: YES), it newly registers thedesignated copy pair in the pair management table 7. Specifically, as inthe case at step SP24, the pair number of this copy pair, LDEV number ofthe virtual LDEVs 53, 76 or internal LDEVs 62, 77 to become the primaryvolume, LDEV number of the virtual LDEVs 53, 76 or internal LDEVs 62, 77to become the secondary volume and the pair status (“split”) arerespectively added to the pair management table 7 (SP27). And, thechannel adapter 70 thereafter creates differential bit information 6associated with this copy pair, and sets “1” to all bits in thisdifferential bit information 6 (SP28). The channel adapter 70 thereafterends the series of processing steps.

(2-2-3) Flow of Load Balancing Processing

Next, the sequential flow of the load balancing processing performedwith this storage system 40 is explained with reference to FIG. 24 toFIG. 35.

The CPU 80 of the management server 44, based on the load balancingsoftware 84, periodically executes the first load balancing processingsequence RT4 shown in FIG. 24, checks the load balance between theexisted storage system 42 and additional storage device 43 relating tothe input and output of data to and from the host system 41, andexecutes processing (load balancing processing) for migrating the LDEVin the existed storage system 42 to the additional storage device 43 toan extent that the load balance of the existed storage system 42 andadditional storage device 43 will not collapse when the load of theexisted storage system 42 is significant.

Here, as the LDEV to be migrated, a LDEV that can be easily migrated ispreferentially used. Specifically, the CPU 80 of the management server44 controls the existed storage system 42, additional storage device 43and external storage device 47 so as to migrate LDEVs in the order of acopy pair which is a virtual LDEV 53 where a copy pair is not formed inthe existed storage system 42 (SP30), a copy pair which is a virtualLDEV 53 where the primary volume and secondary volume are both mapped tothe external LDEV 52 and the pair status is “pair” (SP31), a copy pairwhich is a virtual LDEV 53 where the primary volume and secondary volumeare both mapped to the external LDEV 52 and the pair status is “copy”(SP32), a copy pair which is a virtual LDEV 53 where the primary volumeand secondary volume are both mapped to the external LDEV 52 and thepair status is “split” (SP33), and a copy pair where the primary volumeis the internal LDEV 62 and the secondary volume is the virtual LDEV 53mapped to the external LDEV 52 (SP34).

This kind of load balancing processing is now explained in detail.

(2-2-4) Processing at Step SP30 of First Load Balancing ProcessingSequence RT4

Foremost, the processing contents of the data migration processing formigrating a virtual LDEV 53, in which a pair is not formed in theexisted storage system 42, to the additional storage device 53 to beperformed at step SP30 of the first load balancing processing sequenceRT4 (FIG. 24) are explained. This data migration processing is performedaccording to the first LDEV migration processing sequence RT5 shown inFIG. 25 based on the load balancing software 84.

In other words, when the CPU 80 of the management server 44 proceeds tostep SP30 of the first load balancing processing sequence RT4 shown inFIG. 24, it starts the first LDEV migration processing sequence RT5shown in FIG. 25, foremost accesses the existed storage system 42 andadditional storage device 43 via the LAN 48 (FIG. 13), and acquires theLDEV management table 85 (FIG. 15) and pair management table 7 (FIG. 15)stored in the respective shared memories (FIG. 14) of the existedstorage system 42 and additional storage device 43 (SP40).

Next, the CPU 80 checks the current load balance between the existedstorage system 42 and additional storage device 43 (SP41). Thereupon,when the load of the existed storage system 42 is smaller than the loadof the additional storage device 43 (SP42: NO), there is no need tomigrate the internal LDEV 62 or virtual LDEV 53 of the existed storagesystem 42 to the additional storage device 43. Thus, here, the CPU 80ends this load balancing processing (SP43).

Meanwhile, when the load of the existed storage system 42 is greaterthan the load of the additional storage device 43 (SP42: YES), it isnecessary to migrate the data in the existed storage system 42 to theadditional storage device 43. Thus, here, the CPU 80 searches for thevirtual LDEV 53 that is not paired with another internal LDEV 62 orvirtual LDEV 53 and mapped to the external LDEV 52 set in the externalstorage device 47 among the virtual LDEVs 53 set in the existed storagesystem 42 based on the LDEV management table 85 and pair managementtable 7 of the existed storage system 42 acquired at step SP40 (SP44).

And, when the CPU 80 is not able to detect a virtual LDEV 53 satisfyingthe foregoing conditions as a result of the search (SP45: External LDEVNot Present), it proceeds to step SP31 of the first load balancingprocessing sequence RT4 described above with reference to FIG. 24, and,when it detects a virtual LDEV 53 satisfying the foregoing conditions(SP45: External LDEV Present), it controls the additional storage device43 and creates a virtual LDEV (this is hereinafter referred to as a“substitute virtual LDEV”) 76 having the same capacity as the virtualLDEV (this is hereinafter referred to as a “detected virtual LDEV”) 53detected at step SP44. Further, the CPU 80 switches the mappingdestination of the external LDEV 52 mapped to the detected virtual LDEV53 theretofore to the substitute virtual LDEV 76 (SP47).

And, the CPU 80 thereafter similarly repeats the routine from the tableacquisition processing of step SP40 to the path switching processing ofthe step SP47 (SP40 to SP47), and sequentially migrates the virtualLDEVs 53 not paired with another internal LDEV 62 or virtual LDEV 53 andmapped to the external LDEV 52 among the virtual LDEVs 53 set in theexisted storage system 42 to the additional storage device 43 in LDEVunits to the extent where the load of the additional storage device 43does not exceed the load of the existed storage system 42.

Here, FIG. 26 is a flowchart showing the specific processing contents ofthe CPU 80 in relation to the table acquisition processing describedabove regarding step SP40 of the first LDEV migration processingsequence RT40. The CPU 80 acquires the LDEV management table 85 and pairmanagement table 7 from the existed storage system 42 and additionalstorage device 43 according to the table acquisition processing sequenceRT6 shown in FIG. 26.

In other words, when the CPU 80 proceeds to step SP40 of the first LDEVmigration processing sequence RT5, it starts the table acquisitionprocessing sequence RT6, and foremost transmits a “getLdevTable” command(c.f. FIG. 19) to the existed storage system 42 via the LAN 48 (FIG.13). As a result, data of the LDEV management table 85 is transmittedfrom the existed storage system 42 to the management server 44 accordingto the “getLdevTable” command. When the CPU 80 receives this data, itstores the data in an internal memory not shown (SP51).

Further, the CPU 80 thereafter transmits a “getPairTable” command (c.f.FIG. 20) to the existed storage system 42 via the LAN 48. As a result,data of the pair management table 7 is transmitted from the existedstorage system 42 to the management server 44 according to this“getPairTable” command. And, when the CPU 80 receives this data, itstores the data in the foregoing internal memory (SP52).

Similarly, the CPU 80 thereafter sequentially transmits a “getLdevTable”command and “getPairTable” command to the additional storage device 43via the LAN 48. As a result, data of the management table 85 and data ofthe pair management table 7 are sequentially transmitted to themanagement server 44 according to the “getLdevTable” command and“getPairTable” command. And, when the CPU 80 receives such data, itsequentially stores the data in the foregoing internal memory (SP53,SP54).

Further, FIG. 27 is a flowchart showing the specific processing contentsof the CPU 80 in relation to the load balance check processing regardingstep SP41 of the first LDEV migration processing sequence RT5 (FIG. 25).The CPU 80 quantifies the respective loads of the existed storage system42 and additional storage device 43 (decrease in processing speedaccompanying the increase in the input and output of data) according tothe load balance check processing sequence RT7 shown in FIG. 27, anddetermines which load is higher by comparing these values.

In other words, when the CPU 80 proceeds to step SP41 of the first LDEVmigration processing sequence RT5 (FIG. 25), it starts the load balancecheck processing sequence RT7, and foremost detects the number of copypairs N(P) set in the existed storage system 42 by counting the numberof copy pairs registered in the pair management table 7 of either theexisted storage system 42 or additional storage device 43 (existedstorage system 42 will be used here) (SP60).

Further, the CPU 80 thereafter calculates the number of LDEVs (this ishereinafter referred to as a “normal volume”) N(L) not forming a copypair among the internal LDEVs 62 and virtual LDEVs 53 in the existedstorage system 42 (SP61). Specifically, the CPU 80 counts the number ofentries N(L) in which the LUN is stored in the LUN column 94 of theinternal (virtual) LDEV field 90 (FIG. 16) in the LDEV management table85 of the existed storage system 42, and calculates the differencebetween such count value and twice the number of copy pairs N(P)detected at step SP60.

Next, the CPU 80 separates the internal LDEV 62 and virtual LDEV 53 setin the existed storage system 42 into three types; namely, a LDEVforming a primary volume of a copy pair, a LDEV forming a secondaryvolume of a copy pair, and a normal volume, and quantifies the load L1of the existed storage system 42 by seeking the total value obtained bymultiplying to the number of LDEVs of the respective types a loadingcoefficient predetermined in relation to such types (SP62).

This operation can be calculated with the following formula where theloading coefficient to the primary volume of the copy pair is P, theloading coefficient to the secondary volume of the copy pair is S, andthe loading coefficient to the normal volume is T:L1=N(P)×P+N(P)×S+N(L)  Formula 1

For example, the loading coefficient P to the primary volume of the copypair is set to 1.0, the loading coefficient S to the secondary volume ofthe copy pair is set to 0.5, and the loading coefficient T to the normalvolume is set to 1.0. Incidentally, when the pair status of the copypair is “split”, there is no processing for writing data in thesecondary volume, and the load of the secondary volume will bealleviated. Thus, although the loading coefficient S to the secondaryvolume of the copy pair may be set to be around 0.2 in comparison to theloading coefficient P to the primary volume of the copy pair being setto 1.0, if the primary volume and secondary volume are to beresynchronized, processing for writing data in the secondary volume willbecome required and this could become a high load. This is why theloading coefficient to the secondary volume is set to 0.5 as describedabove.

And, when the number of copy pairs in the existed storage system 42 is400, and the number of normal volumes is 200, the load L1 of the overallexisted storage system 42 can be calculated with the following formula:L1=400×1.0+400×0.5+200×1.0=800  Formula 2

After the operation of the load L1 of this existed storage system 42,the CPU 80 detects the number of copy pairs N (P)′ set in the additionalstorage device 43 by counting the number of copy pairs registered in thepair management table 7 of the additional storage device 43, which isthe data migration destination (SP63).

Next, the CPU 80 calculates the number of normal volumes N (L)′ amongthe internal LDEVs 77 and virtual LDEVs 76 set in the additional storagedevice 43 according to the operation method described above at step SP61(SP64), and thereafter quantifies the load L3 of the additional storagedevice 43 according to the operation method described above at step SP62(SP65).

Next, the CPU 80 determines whether or not the load L1 quantified withthe existed storage system 42 is greater than the load L3 quantifiedwith the additional storage device 43 (SP66), and, when it obtains anegative result (SP66: NO), it obtains a check result of “False(Incorrect)” (SP67), and, when it obtains a positive result (SP66: YES),it obtains a check result of “True (Correct)” (SP68).

As a result, when the CPU 80 obtains a check result of “False(Incorrect)”, it ends the subsequent load balancing processing uponobtaining a negative result in the determination at step SP42 of thefirst LDEV migration processing sequence RT5 shown in FIG. 25 (SP43).Meanwhile, when the CPU 80 obtains a check result of “True (Correct)”,it thereafter proceeds to step SP44 upon obtaining a positive result inthe determination at step SP42 of the first LDEV migration processingsequence RT5.

Meanwhile, FIG. 28 is a flowchart showing the specific processingcontents of step SP44 of the first LDEV migration processing sequenceRT5 (FIG. 25). The CPU 80 searches for a virtual LDEV 53 that is notforming a copy pair and which is mapped to the external LDEV 52according to the first LDEV search processing sequence RT8 shown in FIG.28.

In other words, when the CPU 80 proceeds to step SP44 of the first LDEVmigration processing sequence RT5, it starts this first LDEV searchprocessing sequence RT8, and foremost searches for the internal LDEV 62or virtual LDEV 53 not yet subject to the processing of step SP71 andstep SP73 described later from the LDEV management table 85 of theexisted storage system 42 (SP70).

Next, the CPU 80 refers to the LDEV management table 85 of the existedstorage system 42, and selects an entry from the unchecked internal LDEV62 or virtual LDEV 53 detected at step SP70 in which the port addressand LUN are respectively registered in the port address column 96 andLUN column 97 of the external LDEV field 91 (FIG. 16); that is, itselects one entry of the virtual LDEV 53 mapped to the external LDEV 52in the external storage device 47 (SP71).

When the CPU 80 is not able to select a corresponding entry (SP71:None), it returns to the first LDEV migration processing sequence RT5(SP72), and thereafter proceeds to step SP46 via step SP45 of the firstLDEV migration processing sequence RT5.

Meanwhile, when the CPU 80 is able to select a corresponding entry atstep SP71 (SP71: Present), it searches whether the LDEV number of theinternal LDEV 62 or virtual LDEV 53 associated with such entry isregistered in the pair management table 7 of the existed storage system42 as the LDEV number of the primary volume or the LDEV number of thesecondary volume (SP73).

And, when the CPU 80 is able to detect the LDEV number of the internalLDEV 62 or virtual LDEV 53 associated with the entry selected at stepSP71 in the pair management table 7 of the existed storage system 42 asa result of the foregoing search (SP73: Present), it returns to stepSP70 and repeats the same processing steps (SP70 to SP73-SP70).

Contrarily, when the CPU 80 is not able to detect the LDEV number of theinternal LDEV 62 or virtual LDEV 53 associated with the entry selectedat step SP71 in the pair management table 7 of the existed storagesystem 42 as a result of the foregoing search (SP73: None), it returnsto the first LDEV migration processing sequence RT5 (SP74), andthereafter proceeds to step SP47 via step SP45 of the first LDEVmigration processing sequence RT5.

Meanwhile, FIG. 29 is a flowchart showing the specific processingcontents of the CPU 80 in relation to the path switching processingdescribed above with reference to step SP47 of the first data migrationprocessing (FIG. 25). The CPU 80 executes the foregoing path switchingprocessing according to the path switching processing sequence RT9 shownin FIG. 29.

In other words, when the CPU 80 proceeds to step SP47 of the first LDEVmigration processing sequence RT5, it starts the path switchingprocessing sequence RT9, and foremost transmits an “assignLUN” command(c.f. FIG. 19) to the external storage device 47 (FIG. 13) (SP80). This“assignLUN” command is a command for instructing the destination of thecommand to assign the LDEV of the designated LDEV number to theassignment destination designated via the designated port, and includesthe LDEV number, port address of the destination of the command to beassigned to the assignment destination, and the port address of theassignment destination as arguments.

As a result, when the external storage device 47 receives this“assignLUN” command, it assigns the external LDEV 52 of the designatedLDEV number to the designated external connection port 43B (FIG. 13) ofthe additional storage device 43 via the designated port 47A (FIG. 13).Thereby, the additional storage device 43 will be able to access theexternal LDEV 52. Further, the external storage device 47 notifies theLUN number of this external LDEV 52 to the management server 44 via theLAN 48.

Next, the CPU 80 transmits a “createMapping” command to the additionalstorage device (SP81). Here, the CPU 80 designates the entry information(LDEV number, capacity, port address and LUN of the external storagedevice) registered in the LDEV management table 85 of the internal LDEV62 and virtual LDEV 53 subject to data migration in the existed storagesystem 42 as arguments.

When the additional storage device 43 receives this “createmapping”command, it creates a virtual LDEV 76 (this is hereinafter referred toas a “migration destination LDEV”) having the designated LDEV number anddesignated capacity (same capacity as the internal LDEV 62 or virtualLDEV 53 subject to data migration (this is hereinafter referred to as a“migration source LDEV”), and respectively stores the LDEV number andcapacity of the migration destination LDEV, port address of the externalconnection port 42B (FIG. 13), port address of the external storagedevice acquires at step SP and LUN number of the corresponding externalLDEV 52 in the corresponding columns of the LDEV management table 85.Thereby, the newly created migration destination virtual LDEV isregistered in the LDEV management table 85, and, simultaneouslytherewith, the external LDEV 52 is mapped to the migration destinationvirtual LDEV.

Thereafter, the CPU 80 transmits an “assignLUN” command to theadditional storage device 43. As a result, the additional storage device43 assigns the LUN number of the migration destination virtual LDEV tothe port (HBA 50) of the designated host system 41 via the designatedport 43A. Further, the additional storage device 43 stores this LUNnumber in the LUN column 94 of the internal (virtual) LDEV field 90(FIG. 16) in an entry corresponding to the migration destination virtualLDEV in the LDEV management table 85 (SP82). Thereby, the host system 41will be able to access the migration destination virtual LDEV.

Next, the CPU 80 transmits a deleteLUN” command (c.f. FIG. 19) targetingthe migration source virtual volume to the existed storage system 42.When the existed storage system 42 receives this “deleteLUN” command, itdelete the association between the external LDEV 52 and migration sourcevirtual LDEV by deleting the LUN stored in the LUN column 94 of theinternal (virtual) LDEV field 90 in an entry corresponding to themigration source virtual LDEV among the entries registered in the LDEVmanagement table 85. As a result, the external LDEV 52 will becomeinaccessible via the existed storage system 43.

Thereafter, the CPU 80 transmits a “deletemapping” command to theexisted storage system 42. And, the existed storage system 42 deletesthe association between the external LDEV 52 and migration sourcevirtual LDEV by erasing the data stored in the port address column 96(FIG. 16) and LUN column 97 (FIG. 16) of the external LDEV field 91(FIG. 16) in an entry corresponding to the migration source virtual LDEVin the LDEV management table 85 based on this “deleteMapping” command,and thereafter deletes the migration source virtual LDEV by erasing theentry corresponding to the migration source virtual LDEV in the LDEVmanagement table 85.

Incidentally, in the foregoing case, the LUN number of the external LDEV52 in which the mapping was deleted, port address of the port 47A (FIG.13) of the external storage device 47, and port address of the externalconnection port 42B (FIG. 13) in the existed storage system 42 willstill be retained by the load balancing software 84 (FIG. 13).

Next, the CPU 80 transmits a “deleteLUN” command for designating the LUNnumber of the external LDEV 52, port address of the port 47A of theexternal storage device 47, and port address of the external connectionport 42B to the external storage device 47. And, when the externalstorage device 47 receives this “deleteLUN” command, it deletes theassignment of the external LDEV 52 to the existed storage system 42according to this “deleteLUN” command.

(2-2-5) Processing at Step SP31 of First Load Balancing ProcessingSequence RT4

Next, the processing of migrating the copy pair in which the primaryvolume and secondary volume are both mapped to the external LDEV 52(FIG. 13) and in which the pair status is “pair” to the additionalstorage device 43 to be performed at step SP31 of the first loadbalancing processing sequence RT4 (FIG. 24) is explained. This migrationprocessing is performed according to the second LDEV migrationprocessing sequence RT10 shown in FIG. 30 based on the load balancingsoftware 84 (FIG. 13).

In other words, when the CPU 80 of the management server 44 proceeds tostep SP31 of the first load balancing processing sequence RT4 describedabove with reference to FIG. 24, it starts the second LDEV migrationprocessing sequence RT10 shown in FIG. 30, and foremost acquires theLDEV management table 85 and pair management table 7 of the existedstorage system 42 and additional storage device 43 according to thetable acquisition processing sequence RT6 described above with referenceto FIG. 26 (SP90).

Next, the CPU 80 checks the current load balance between the existedstorage system 42 and additional storage device 43 according to the loadbalance check processing sequence RT7 described above with reference toFIG. 27 (SP91). As a result, when the load of the existed storage system42 is smaller than the load of the additional storage device 43 (SP92:NO), the CPU 80 ends this load balancing processing (SP93).

Meanwhile, when the load of the existed storage system 42 is greaterthan the load of the additional storage device 43 (SP92: YES), the CPU80 searches for the copy pair in which the primary volume and secondaryvolume are both mapped to the external LDEV 52 in the external storagedevice 47 and in which the pair status is “pair” based on the LDEVmanagement table 85 and pair management table 7 of the existed storagesystem 42 acquired at step SP90 (SP94).

When the CPU 80 does not detect a copy pair satisfying the foregoingconditions as a result of the foregoing search (SP95: NO), it proceedsto step SP32 of the first load balancing processing sequence RT4 (FIG.24) (SP96), and, contrarily, when the CPU 80 detects a copy pairsatisfying the foregoing conditions (SP95: YES), it searches for a LDEVnumber of the primary volume configuring this copy pair from the LDEVmanagement table 85 of the existed storage system 42 (SP97).

And, the CPU 80 thereafter migrates the primary volume of the copy pairdetected at step SP94 to the additional storage device 43 according tothe path switching processing sequence RT9 described above withreference to FIG. 29 based on the foregoing search result (SP98).

Further, the CPU 80 searches for a LDEV number of the secondary volumeconfiguring the copy pair detected at step SP94 from the LDEV managementtable 85 of the existed storage system 42 (SP99), and thereaftermigrates the secondary volume to the additional storage device 43according to the path switching processing sequence RT9 described abovewith reference to FIG. 29 based on the foregoing search result (SP100).

Next, the CPU 80 transmits a “deletepair” command to the existed storagesystem 42, and deletes the copy pair from the existed storage system 42by deleting the entry of the pair number corresponding to the copy pairdetected at step SP94 registered in the pair management table 7 of theexisted storage system 42 (SP101).

Further, the CPU 80 thereafter transmits a “createpair” command formaking the pair status of the copy pair to be created “pair”; that is,it transmits a “createpair -pair” to the additional storage device 43,and sets “pair” as the pair status of the copy pair migrated to theadditional storage device 43, and sets “0” as the value of all bits ofthe differential bit information 6 of the copy pair (SP102). Thereby,since the pair status differential bit information that is the same asthe pair status and differential bit information 6 of the copy pair setin the existed storage system 42 before being migrated to the additionalstorage device 43 will be set in the additional storage device 43, thecopy status of the copy pair before migration will be consequentlysucceed by the additional storage device 43 together with the migrationof the copy pair.

And, the CPU 80 thereafter repeats the same processing steps (SP90 toSP102), and thereby sequentially migrates the copy pairs in which theprimary volume and secondary volume are both mapped to the external LDEV52 and the pair status is “pair” to the additional storage device 43.

Here, FIG. 31 is a flowchart showing the processing contents forsearching the copy pair in which the primary volume and secondary volumeare both mapped to the external LDEV 52 and the pair status is “pair”described with reference to step SP94 of the second data migrationprocessing sequence RT10 (FIG. 30).

When the CPU 80 proceeds to step SP94 of the second LDEV migrationprocessing sequence RT10, it starts the first copy pair searchprocessing sequence RT11 shown in FIG. 31, and foremost searches anentry of the copy pair of the designated pair status (“pair”) from thepair status column 103 (FIG. 17) in the pair management table 7 of theexisted storage system 42 (SP110).

And, when the CPU 80 is not able detect an entry of the designated pairstatus as a result of the foregoing search (SP110: None), it returns tothe second LDEV migration processing sequence RT10 (SP117), andthereafter proceeds to step SP32 of the first load balancing processingsequence RT4 via step SP95 of the second LDEV migration processingsequence RT10.

Meanwhile, when the CPU 80 is able to an entry of the designated pairstatus as a result of the foregoing search (SP110: Present), it acquiresthe LDEV number of the primary volume of the detected entry from thepair management table 7 of the existed storage system 42 (SP111), andthereafter searches for an entry storing the LDEV number acquired atstep SP111 from the LDEV number stored in the respective LDEV numbercolumns 92 (FIG. 16) of the internal (virtual) LDEV field 90 (FIG. 16)in the LDEV management table 85 (FIG. 16) of the existed storage system42 (SP112).

And, when the CPU 80 detects this entry, it confirms whether or not thecorresponding port address and the LUN of the external LDEV 52 arestored in the port address column 96 (FIG. 16) and LUN column 97 (FIG.16) of the external LDEV field 91 (FIG. 16) of the entry; that is,whether the external LDEV 52 is mapped to the virtual LDEV 53 in theexisted storage system 42 corresponding to such entry (SP113).

And, when the CPU 80 obtains a negative result in the confirmation atstep SP113 (SP113: NO), it returns to step SP110, and, contrarily, whenthe CPU 80 obtains a positive result (SP113: YES), it acquires the LDEVnumber of the secondary volume of the entry detected at step SP110 fromthe pair management table 7 of the existed storage system 42 (SP114).

Further, the CPU 80 thereafter searches for an entry storing the LDEVnumber acquired at step SP114 from the LDEV number column 92 (FIG. 16)of the internal (virtual) LDEV field 90 (FIG. 16) in the LDEV managementtable 85 of the existed storage system 42 (SP115).

Further, when the CPU 80 detects a corresponding entry as a result ofthe search at step SP115, it confirms whether the corresponding portaddress and LUN of the external LDEV 52 are stored in the port addresscolumn 96 (FIG. 16) and LUN column 97 (FIG. 16) in the external LDEVfield 91 (FIG. 16) of the entry; that is, whether the external LDEV 52is mapped to the LDEV in the existed storage system 42 corresponding tothis entry (SP116).

And, when the CPU 80 obtains a negative result in the confirmation atstep SP116 (SP116: NO), it returns to step SP110, and, contrarily, whenthe CPU 80 obtains a positive result (SP116: YES), it returns to thesecond LDEV migration processing sequence RT10 (SP118), and thereafterproceeds to step SP97 of the second LDEV migration processing sequenceRT10.

(2-2-6) Processing at Step SP32 of First Load Balancing ProcessingSequence RT4

Next, the processing of migrating the copy pair in which the primaryvolume and secondary volume are both mapped to the external LDEV 52 andin which the pair status is “copy” to the additional storage device 43to be performed at step SP32 of the first load balancing processingsequence RT4 (FIG. 24) is explained. This migration processing isperformed according to the third LDEV migration processing sequence RT12shown in FIG. 32 based on the load balancing software 84 (FIG. 13).

In other words, when the CPU 80 of the management server 44 proceeds tostep SP32 of the first load balancing processing sequence RT4 describedabove with reference to FIG. 24, it starts the third LDEV migrationprocessing sequence RT12 shown in FIG. 32, and foremost acquires theLDEV management table 85 and pair management table 7 of the existedstorage system 42 and additional storage device 43 according to thetable acquisition processing sequence RT6 described above with referenceto FIG. 26 (SP120).

Next, the CPU 80 checks the current load balance between the existedstorage system 42 and additional storage device 43 according to the loadbalance check processing sequence RT7 described above with reference toFIG. 27 (SP121). As a result, when the load of the existed storagesystem 42 is smaller than the load of the additional storage device 43(SP122: NO), the CPU 80 ends this load balancing processing (SP123).

Meanwhile, when the load of the existed storage system 42 is greaterthan the load of the additional storage device 43 (SP92: YES), the CPU80 searches for the copy pair in which the primary volume and secondaryvolume are both mapped to the external LDEV 52 in the external storagedevice 47 and in which the pair status is “copy” based on the LDEVmanagement table 85 and pair management table 7 of the existed storagesystem 42 acquired at step SP120 (SP124). Incidentally, the processingat step SP124 may be performed according to the first copy pair searchprocessing sequence RT11 in which the designated pair status at stepSP110 of the first copy pair search processing sequence RT11 describedabove with reference to FIG. 31 is “copy”.

When the CPU 80 does not detect a copy pair satisfying the foregoingconditions as a result of the foregoing search (SP125: NO), it proceedsto step SP33 of the first load balancing processing sequence RT4(SP126), and, contrarily, when the CPU 80 detects a copy pair satisfyingthe foregoing conditions (SP125: YES), it stands by until the copyprocessing of this copy pair is complete. Specifically, the CPU 80issues a “getLdevTable” command periodically (e.g. once per minute) tothe existed storage system 42 until the pair status of this copy pairbecomes “pair”, and repeats the processing for confirming the pairstatus of the copy pair.

And, when the pair status of the copy pair eventually becomes “pair”,the CPU 80 executes the processing of step SP128 to SP133 as with theprocessing of step SP97 to SP102 of the second LDEV migration processingsequence RT10 described above with reference to FIG. 30, and therebymigrates such copy pair to the additional storage device 43, and makesthe copy pair migrated to the additional storage device 43 succeed themanagement information (pair status and differential bit information 6)of the copy pair.

And, the CPU 80 thereafter repeats the same processing steps (SP120 toSP133), and thereby sequentially migrates the copy pairs in which theprimary volume and secondary volume are both mapped to the external LDEV52 and the pair status is “copy” to the additional storage device 43.

Incidentally, although the foregoing description explained a case wherethe CPU 80 stood by until the copy processing of the copy pair wascompleted (SP127), and thereafter switch paths to form a copy pair wherethe pair status is “pair” (SP128 to SP133), the CPU 80 may alsodiscontinue the copy processing being performed without waiting for thecopy processing of the copy pair to be completed, switching the path ofthe primary volume and secondary volume, and thereafter (after executingthe processing at step SP128 to SP131) adding an entry to the pairmanagement table 85 of the additional storage device 43, and set “1” toall bits of the corresponding differential bit information 6 in order toimplement copying (making the pair status “copy”).

(2-2-7) Processing at Step SP33 of First Load Balancing ProcessingSequence RT4

Next, the processing of migrating the copy pair in which the primaryvolume and secondary volume are both mapped to the external LDEV 52(FIG. 13) and in which the pair status is “split” to the additionalstorage device 43 to be performed at step SP33 of the first loadbalancing processing sequence RT4 (FIG. 24) is explained. This migrationprocessing is performed according to the fourth LDEV migrationprocessing sequence RT13 shown in FIG. 33 based on the load balancingsoftware 84 (FIG. 13).

In other words, when the CPU 80 of the management server 44 proceeds tostep SP33 of the first load balancing processing sequence RT4 describedabove with reference to FIG. 24, it starts the fourth LDEV migrationprocessing sequence RT13 shown in FIG. 33, and foremost acquires theLDEV management table 85 and pair management table 7 of the existedstorage system 42 and additional storage device 43 according to thetable acquisition processing sequence RT6 described above with referenceto FIG. 26 (SP140).

Next, the CPU 80 checks the current load balance between the existedstorage system 42 and additional storage device 43 according to the loadbalance check processing sequence RT7 described above with reference toFIG. 27 (SP141). As a result, when the load of the existed storagesystem 42 is smaller than the load of the additional storage device 43(SP142: NO), the CPU 80 ends this load balancing processing (SP143).

Meanwhile, when the load of the existed storage system 42 is greaterthan the load of the additional storage device 43 (SP142: YES), the CPU80 searches for the copy pair in which the primary volume and secondaryvolume are both mapped to the external LDEV 52 in the external storagedevice 47 and in which the pair status is “split” based on the LDEVmanagement table 85 and pair management table 7 of the existed storagesystem 42 acquired at step SP140 (SP144).

When the CPU 80 does not detect a copy pair satisfying the foregoingconditions as a result of the foregoing search (SP145: NO), it proceedsto step SP34 of the first load balancing processing sequence RT4 (FIG.24) (SP146), and, contrarily, when the CPU 80 detects a copy pairsatisfying the foregoing conditions (SP145: YES), it thereafter executesthe processing of step SP146 to SP150 as with the processing of stepSP97 to SP101 of the second LDEV migration processing sequence RT10described above with reference to FIG. 30, and thereby migrates the copypair and the management information (pair status, etc.) thereof to theadditional storage device 43.

Further, the CPU 80 thereafter transmits a “createpair” command formaking the pair status of the copy pair to be created “split”; that is,it transmits a “createpair-split” to the additional storage device 43,and sets “split” as the pair status of the copy pair migrated to theadditional storage device 43, and sets “1” as the value of all bits ofthe differential bit information 6 of the copy pair (SP151). Thereby,the pair status that is the same as the pair status of the copy pair setin the existed storage system 42 before being migrated to the additionalstorage device 43 will be set in the additional storage device 43.Further, although the differential bit information 6 storing thedifference location of the primary volume and secondary volume beforethe copy pair was migrated to the additional storage device 43 exists inthe existed storage system 42, since “1” is set to all bits of thedifferential bit information 6 of the additional storage device 43, evenif there is no differential bit information 6 before the migration (evenif the differential bit information is not succeeded by the additionalstorage device 43), loss of data in the logical device will not occurwhen synchronizing the copy pair (“resyn” command, etc.) after migratingsuch copy pair to the additional storage device 43.

And, the CPU 80 thereafter repeats the same processing steps (SP140 toSP151), and thereby sequentially migrates the copy pairs in which theprimary volume and secondary volume are both mapped to the external LDEV52 and the pair status is “split” to the additional storage device 43.

(2-2-8) Processing at Step SP34 of First Load Balancing ProcessingSequence RT4

Next, the processing performed at step SP34 of the first load balancingprocessing sequence RT4 (FIG. 24) is explained. At step SP34, aninternal mirror copy pair is formed, and processing for migrating a copypair in which the primary volume is an internal LDEV 62 and thesecondary volume is a virtual LDEV 53 mapped to the external LDEV 52 tothe additional storage device 43 is explained. And, this migrationprocessing is performed according to the fifth LDEV migration processingsequence RT14 shown in FIG. 35 based on the load balancing software 84(FIG. 13).

In other words, when the CPU 80 of the management server 44 proceeds tostep SP34 of the first load balancing processing sequence RT4 describedabove with reference to FIG. 24, it starts the fifth LDEV migrationprocessing sequence RT14 shown in FIG. 34, and foremost acquires theLDEV management table 85 and pair management table 7 of the existedstorage system 42 and additional storage device 43 according to thetable acquisition processing sequence RT6 described above with referenceto FIG. 26 (SP160).

Next, the CPU 80 checks the current load balance between the existedstorage system 42 and additional storage device 43 according to the loadbalance check processing sequence RT7 described above with reference toFIG. 27 (SP161). As a result, when the load of the existed storagesystem 42 is smaller than the load of the additional storage device 43(SP162: NO), the CPU 80 ends this load balancing processing (SP163).

Meanwhile, when the load of the existed storage system 42 is greaterthan the load of the additional storage device 43 (SP162: YES), the CPU80 searches for the copy pair which is a virtual LDEV 53 formed in aninternal mirror copy pair and in which the primary volume is an internalLDEV 62 and the secondary volume is a virtual LDEV 53 mapped to theexternal LDEV 52 (SP163).

When the CPU 80 does not detect a copy pair satisfying the foregoingconditions as a result of the foregoing search (SP164: NO), it end thisload balancing processing (SP165), and, contrarily, when the CPU 80detects a copy pair satisfying the foregoing conditions (SP164: YES), itsearches the LDEV management table 85 of the existed storage system 42and confirms the current pair status (“pair”, “copy” or “split”) of thecopy pair (SP166).

The CPU 80 proceeds to step SP169 when the pair status of the copy pairis not “copy” (SP167: NO), and, contrarily, when the pair status of thecopy pair is “copy” (SP167: YES), it stands by until the copy processingof the copy pair is complete (SP168). Specifically, as with step SP127of the third LDEV migration processing sequence RT12 described withreference to FIG. 32, the CPU 80 issues a “getLdevTable” commandperiodically (e.g. once per minute) to the existed storage system 42until the pair status of this copy pair becomes “pair”, and repeats theprocessing for confirming the pair status of the copy pair.

And, when the copy processing of the copy pair is eventually completedand the pair status of the copy pair becomes “pair”, the CPU 80 createsan internal LDEV 77 (FIG. 13) having the same capacity as the primaryvolume of the copy pair detected at step SP163 in the additional storagedevice 43 by issuing a “createLdev” command to the additional storagedevice 43 (SP170).

Further, the CPU 80 assigns the internal LDEV 77 created in theadditional storage device 43 at step SP169 to the host system 41 (FIG.13) by issuing an “assignLUN” command to the additional storage device43. As a result of this processing, the host system 41 will be able toaccess the internal LDEV 77.

Thereafter, the CPU 80 uses a SCSI (Small Computer System Interface)command or the like and copies the data of the primary volume in theexisted storage system 42 in block units to the internal LDEV 77 in theadditional storage device 43 created at step SP169 (SP171).

Further, the CPU 80 migrates the secondary volume to the additionalstorage device 43 and thereafter deletes the copy pair detected at stepSP163 from the existed storage system 42 by executing the sameprocessing as step SP99 to step SP101 of the second LDEV migrationprocessing sequence RT10 of FIG. 30 (SP172 to SP174).

Further, the CPU 80 thereafter creates a copy pair formed from a primaryvolume migrated to the additional storage device 43 with the processingof step SP169 to step SP172 and a secondary volume migrated to theadditional storage device 43 with the processing of step SP172 and stepSP173 in the additional storage device 43 by issuing a “createpair-init” to the additional storage device 43. Thereby, the pair status anddifferential bit information 6 of the copy pair before being migrated tothe additional storage device 43 are succeeded by the migrated copy pair(strictly speaking, the same pair status and differential bitinformation before migration are set), and the data copy between theprimary volume and secondary volume is commenced thereafter (SP175).

Here, FIG. 35 is a flowchart showing the processing contents forsearching the copy pair forming an internal mirror copy pair and inwhich the primary volume is an internal LDEV 62 and the secondary volumeis an external LDEV 52 mapped to the virtual LDEV 53 as described withreference to step SP163 of the fifth LDEV migration processing sequenceRT14 (FIG. 34).

When the CPU 80 proceeds to step SP163 of the fifth LDEV migrationprocessing sequence RT14, it starts the second copy pair searchprocessing sequence RT15 shown in FIG. 35, and foremost selects oneprimary volume from the primary volume LDEV number column 101 (FIG. 17)in the pair management table 7 of the existed storage system 42 (SP180).

And, when the CPU 80 is not able to select a primary volume; that is,when a copy pair is not registered in the pair management table 7 of theexisted storage system 42 (SP180: None), it returns to the fifth LDEVmigration processing sequence RT14 described with reference to FIG. 34(SP181), and thereafter proceeds to step SP165 via step SP164 of thefifth LDEV migration processing sequence RT14 so as to end this loadbalancing processing (SP165).

Meanwhile, when the CPU 80 is able to select a LDEV number of a primaryvolume at step SP180 (SP180: Present), it searches for a LDEV# in theLDEV management table 85 (FIG. 16) of the existed storage system 42, anddetects the entry of the LDEV number of such selected primary volume(SP182).

Next, the CPU 80 confirms whether the corresponding port address and LUNof the external LDEV 52 are stored in the port address column 96 (FIG.16) and LUN column 97 (FIG. 16) of the external LDEV field 91 (FIG. 16)of the entry detected at step SP182 in the LDEV management table 85 ofthe existed storage. system 42; that is, whether the external LDEV 52 ismapped to the virtual LDEV 53 in the existed storage system 42corresponding to such entry (SP183).

And, when the CPU 80 obtains a positive result in the foregoingconfirmation (SP183: YES), it returns to step SP180, and, contrarily,when the CPU 80 obtains a negative result (SP183: NO), it acquires theLDEV number of the secondary volume of the copy pair that selected theLDEV number of the primary volume at step SP181 from the pair managementtable 7 (FIG. 15) of the existed storage system 42 (SP184).

Next, the CPU 80 searches for the LDEV# in the LDEV management table 85(FIG. 16) of the existed storage system 42, and detects the entry of theLDEV number of the secondary volume acquired at step SP184 (SP185).

Further, the CPU 80 thereafter confirms whether the corresponding portaddress and LUN of the external LDEV 52 are stored in the port addresscolumn 96 (FIG. 16) and LUN column 97 (FIG. 16) in the external LDEVfield 91 (FIG. 16) of the entry detected at step SP185 in the LDEVmanagement table 85 of the existed storage system 42; that is, whetherthe external LDEV 52 is mapped to the virtual LDEV 62 in the existedstorage system 42 corresponding to such entry (SP186).

And, when the CPU 80 obtains a negative result in the foregoingconfirmation (SP186: NO), it returns to step SP180, and, contrarily,when the CPU 80 obtains a positive result (SP186: YES), it returns tothe fifth LDEV migration processing sequence RT14 described above withreference to FIG. 34 (SP187), and thereafter proceeds to step SP166 viastep SP164 of this fifth LDEV migration processing sequence RT14.

(2-3) Effect of Present Embodiment

In the foregoing configuration, with the storage system 40 of thisembodiment, the management server 44 periodically detects the respectiveloads of the existed storage system 42 and additional storage device 43,and preferentially migrates the internal LDEV 62 or virtual LDEV 53 inwhich a copy pair is not formed in the existed storage system 42 to theadditional storage device 43 when the load of the existed storage system42 is greater than the load of the additional storage device 43. Thus,if the load balancing processing of the existed storage system 43 can beended at this stage, the load balancing processing of the existedstorage system 42 can be performed regardless of the current status ofthe copy pair set in the existed storage system 42.

Meanwhile, if it is not possible to end the load balancing processing ofthe existed storage system 42 at this stage, the storage system 40migrates the virtual LDEVs 53 to the additional storage device 43 byswitching the paths of the respective virtual LDEVs 53 forming the copypair of the primary volume and secondary volume set in the existedstorage system 42 based on the control of the management server 44.Further, the management server 44 thereafter issues a “createpair”command, with an argument (“-init”, “-pair” or “-split”) according tothe pair status before the migration of the primary volume and secondaryvolume added thereto, to the additional storage device 43 based on theLDEV management table 85 and pair management table 7 read from theexisted storage system 42, and the additional storage device 43 sets thepair status and differential bit information 6 of the migrated primaryvolume and secondary volume according to this “createpair” command asthe same pair status and differential bit information of thecorresponding primary volume and secondary volume before migration.

Accordingly, with this storage system 40, even when migrating theprimary volume and secondary volume, which are respectively mapped tothe external LDEV 52 of the external storage device 47 based on theexternal connection function and subject to internal mirror copyingbased on the internal mirror copying function, to the additional storagedevice 43, the pair status and differential bit information 6 of suchcopy pair can be set to be the same as before the migration, or thecurrent status of the copy pair can be set in the additional storagedevice 43 so that there will no difference with the data beforemigration caused by performing the data migration of such copy pair, andload balancing will also be enabled in the foregoing cases.

(3) Second Embodiment

(3-1) Outline of Load Balancing Processing According to SecondEmbodiment

In the load balancing processing according to the first load balancingprocessing sequence RT4 described above with reference to FIG. 24, uponmigrating the copy pair in which the pair status is “copy” at step SP32to the additional storage device 43, time will be wasted since there isa standby period (step SP127 of FIG. 32) until the copy processing ofthe copy pair is completed.

Further, with the load balancing processing according to the first loadbalancing processing sequence RT4, upon migrating the copy pair in whichthe pair status is “split” at step SP33 to the additional storage device43, since the differential bit information 6 (FIG. 18) is not migrated,there is a slight disadvantage in that the copy volume will increaseupon resynchronizing the primary volume and secondary volume additionalmigrated to the storage device 43.

Thus, by collectively processing the copy pair in which the primaryvolume and secondary volume are both mapped to the external LDEV 52 andthe pair status is “copy” or “split”, and the copy pair forming aninternal mirror copy pair and in which the primary volume is theinternal LDEV 62 and the secondary volume is the virtual LDEV 53 mappedto the external LDEV 52 in the same timing without differentiating suchcopy pairs, the load balancing processing can be expedited in comparisonto the load balancing processing according to the first load balancingprocessing sequence RT4.

(3-2) Configuration of Storage System According to Present Embodiment

In FIG. 13, reference numeral 100 represents the overall storage systemaccording to the second embodiment. This storage system 100 isconfigured the same as the storage system 40 (FIG. 13) according to thefirst embodiment excluding the point in that the CPU 80 of themanagement server 101 performs the load balancing processing forbalancing the load of the existed storage system 42 with the additionalstorage device 43 according to the second load balancing processingsequence RT16 shown in FIG. 36 in substitute for the first loadbalancing processing sequence RT4 described above with reference to FIG.24 based on the load balancing software 102 stored in the physicalstorage devices 81.

In actuality, the CPU 80 of the management server 101 periodicallyexecutes this second load balancing processing sequence RT16, and,foremost, as with step SP30 of the first load balancing processingsequence RT4 (FIG. 24), migrates the internal LDEV 62 or virtual LDEV 53in which a copy pair is not formed in the existed storage system 42 tothe additional storage device 53 (SP190), and thereafter, as with stepSP31 of the first load balancing processing sequence RT4, maps both theprimary volume and secondary volume to the external LDEV 52 (FIG. 13),and migrates the copy pair in which the pair status is “pair” to theadditional storage device 43 (SP191).

And, the CPU 80 thereafter migrates a copy pair in the existed storagesystem 42 in which the pair status is not “pair” to the additionalstorage device 43 (SP192).

FIG. 37 and FIG. 38 are flowcharts showing the specific contents of themigration processing of such a copy pair. The CPU 80, upon proceeding tostep SP192 of the second load balancing processing sequence RT16, startsthe sixth LDEV migration processing sequence RT17 shown in FIG. 37 andFIG. 38, and foremost acquires the LDEV management table 85 and pairmanagement table 7 of the existed storage system 42 and additionalstorage device 43 according to the table acquisition processing sequenceRT6 described above with reference to FIG. 26 (SP200).

Next, the CPU 80 checks the current load balance between the existedstorage system 42 and additional storage device 43 according to the loadbalance check processing sequence RT7 described above with reference toFIG. 27 (SP201). As a result, if the load of the existed storage system42 is smaller than the load of the additional storage device 43 (SP202:NO), the CPU 80 ends this load balancing processing (SP203).

Contrarily, if the load of the existed storage system 42 is greater thanthe load of the additional storage device 43 (SP202: YES), the CPU 80selects on entry of a copy pair in which the pair status is not “pair”from the pair management table 7 of the existed storage system 42acquired at step SP200 (SP204).

And, the CPU 80 thereafter confirms the pair status of the copy pairbased on this pair management table 7 (SP205), and, when the confirmedpair status is “copy” (SP206: YES), it stands by until the copyprocessing of this copy pair is completed (SP207).

Meanwhile, when the pair status of the copy pair is not “copy” (SP206:NO), or the pair status of the copy pair is “copy” but the copyprocessing of such copy pair has been completed, the CPU 80 searches fora LDEV number of the primary volume LDEV forming the copy pair from theLDEV management table 85 of the existed storage system 42 (SP208), andthereafter determines whether the primary volume is the virtual LDEV 53mapped to the external LDEV 52 based on such search result (SP209).

When the CPU 80 obtains a positive result in this determination (SP209:YES), it executes the processing for migrating this primary volume tothe additional storage device 43 according to the path switchingprocessing sequence RT9 described above with reference to FIG. 29(SP210).

Meanwhile, when the CPU 80 obtains a negative result in thedetermination at step SP219 (SP209: NO), as with step SP169 to stepSP171 of the fifth LDEV migration processing sequence RT14 describedabove with reference to FIG. 34, it migrates the primary volume to theadditional storage device 43 (SP211 to SP213).

Thereafter, the CPU 80 searches for a LDEV number of a secondary volumeLDEV forming the copy pair selected at step SP214 from the LDEVmanagement table 85 of the existed storage system 42 (SP214), anddetermines whether or not the secondary volume is the virtual LDEV 53mapped to the external LDEV 52 based on this search result (SP215).

When the CPU 80 obtains a positive result in this determination (SP215:YES), it executes processing for migrating this secondary volume to theadditional storage device 43 according to the path switching processingsequence RT9 described above with reference to FIG. 29 (SP216).

Meanwhile, when the CPU 80 obtains a negative result in thedetermination at step SP215 (SP215: NO), it creates a virtual LDEV 76(FIG. 13) having the same capacity as the secondary volume in theadditional storage device 43 by issuing a “createLdev” command to theadditional storage device 43 (SP217).

Next, the CPU 80 deletes the entry of the corresponding pair numberdetected at step SP214 registered in the pair management table 7 of theexisted storage system 42 by transmitting a “deletepair” command to theexisted storage system 42 (SP218).

Further, the CPU 80 thereafter creates a copy pair formed from a primaryvolume migrated to the additional storage device 43 with the processingat step SP209 to step SP213 and the secondary volume migrated to theadditional storage device 43 with the processing at step SP215 to stepSP217 in the additional storage device 43 by issuing a “createpair-nit”command to the additional storage device 43. As a result, the pairstatus of the copy pair and the differential bit information 6 beforemigration to the additional storage device 43 can be succeeded by themigrated copy pair, and the data copy between the primary volume andsecondary volume is commenced thereafter (SP219).

Further, the CPU 80 thereafter returns to step SP210, and repeats thesame processing until the load of the existed storage system 42 becomessmaller than the load of the additional storage device 43 (SP200 toSP219).

According to the load balancing processing according to this kind ofsecond load balancing processing sequence RT16 (FIG. 36), the loadbalancing processing can be expedited in comparison to the loadbalancing processing according to the first load balancing processingsequence RT4. As a result, in addition to the effect yielded in thefirst embodiment, a storage system capable of further expediting theload balancing processing can be realized.

(4) Third Embodiment

In FIG. 13, reference numeral 110 represents the overall storage systemaccording to the third embodiment. This storage system 110 is configuredthe same as the storage system 40 (FIG. 13) according to the firstembodiment excluding the point in that the CPU 80 of the managementserver 111 performs load balancing processing for balancing the load ofthe existed storage system 42 with the additional storage device 43according to the third load balancing processing sequence RT17 insubstitute for the first load balancing processing sequence RT4described above with reference to FIG. 24 based on the load balancingsoftware 112 stored in the physical storage devices 81.

In actuality, the CPU 80 of the management server 101 periodicallyexecutes this third load balancing processing sequence RT18, and, aswith step SP30 of the first load balancing processing sequence RT4 (FIG.24), it migrates the internal LDEV 62 or virtual LDEV 53 in which a copypair is not formed in the existed storage system 42 to the additionalstorage device 53 (SP220), and thereafter ends this load balancingprocessing.

According to the load balancing processing of the third embodiment, incomparison to the storage system 110 of the second embodiment, a storagesystem capable of balancing the load of the existed storage system 42more expeditiously can be realized.

(5) Fourth Embodiment

FIG. 40, in which the same reference numerals are given to the samecomponents as those corresponding to FIG. 13, show the storage system120 according to the fourth embodiment. This storage system 120 isconfigured the same as the storage system 40 of the first embodimentexcluding the point of an installed fibre channel switch 121 and anadditional fibre channel switch 122 equipped with the samevirtualization function as the internal mirror copying function andexternal connection function being provided in substitute for theexisted storage system 42 and additional storage device 43 of thestorage system 40 (FIG. 13) according to the first embodiment, and thepoint of the external storage device 57 being directly connected to theinstalled fibre channel switch 121 and additional fibre channel switch122.

The installed fibre channel switch 121 and additional fibre channelswitch 122, as shown in FIG. 41, are respectively configured by having aplurality of ports 121A, 122A retaining a routing table 123, a CPU 124for controlling the overall operation of the installed fibre channelswitch 121 or additional fibre channel switch 122, a memory 125 storingvarious control programs, and a crossbar switch 126 for connecting theseports 121A, 122A, CPU 124 and memory 125.

The installed fibre channel switch 121 and additional fibre channelswitch 122 normally sort the fibre channel packets transmitted from thehost system 41 and external storage device 57 by delivering them fromthe ports 121A, 122A to the corresponding ports 121A, 122A according tothe routing table 123 retained in the respective ports 121A, 122A, and,upon executing the virtualization function, virtualize the external LDEV52 set in the connected external storage device 57.

As a means for achieving the above, the installed fibre channel switch121 and additional fibre channel switch 122 have a function as a SCSItarget for creating the virtual LDEVs 130, 131 and providing these tothe host system 41, and an initiator function for re-issuing a data I/Orequest to the virtual LDEVs 130, 131 as the data I/O request to theexternal storage device 57 under its control.

And, the installed fibre channel switch 121 and additional fibre channelswitch 122, upon executing the virtualization function, interprets thefibre channel packets transmitted from the host system 41 in the CPU124, and, when this is a data I/O request to the virtual LDEVs 130, 131,it issues the data I/O request to the external LDEV 52 mapped to thesevirtual LDEVs 130, 131 to the corresponding external storage device 57.

Further, the installed fibre channel switch 121 and additional fibrechannel switch 122 retain in the memory 125 the foregoing LDEVmanagement table 85, pair management table 7 and differential bitinformation 6 of each copy pair, and manages the virtual LDEVs 130, 131and copy pairs set in the installed fibre channel switch 121 oradditional fibre channel switch 122 based on the LDEV management table85, pair management table 7 and differential bit information 6.

Meanwhile, the management server 111, as with the first embodiment,executes the load balancing processing for balancing the load of theinstalled fibre channel switch 121 with the additional fibre channelswitch 122 according to the first load balancing processing sequence RT4shown in FIG. 24 based on the LDEV management table 85 and pairmanagement table 7 of the installed fibre channel switch 121 andadditional fibre channel switch 122.

However, in the foregoing case, since an internal LDEV will never be setin the installed fibre channel switch 121 and additional fibre channelswitch 122, step SP34 of the first load balancing processing sequenceRT4 is omitted.

As described above, even in a case where the virtualization means forvirtualizing the external LDEV 52 of the external storage device 57 is afibre channel switch (installed fibre channel switch 121, additionalfibre channel switch 122), the load can be balanced as in the case wherethe virtualization means is a storage device (existed storage system 42,additional storage device 43).

(6) Other Embodiments

Incidentally, in the foregoing first to fourth embodiments, although acase was explained where the present invention is employed in storagesystems 40, 100, 110, 120 where the virtualization means forvirtualizing the external LDEV 52 of the external storage device 57 is astorage device or fibre channel switch, the present invention is notlimited thereto, and may be broadly employed in various storage systemsso as long as the storage system uses, as the virtualization means, anapparatus equipped with a first function for copying data of the primarylogical device paired in itself to the secondary logical device withoutgoing through the host system, and a second function for virtualizingthe external logical device and providing it to the host system.

Further, in the foregoing fourth embodiment, although a case wasexplained where the management server 44 executes processing forbalancing the load of the installed fibre channel switch 121 accordingto the first load balancing processing sequence RT4 (FIG. 24), thepresent invention is not limited thereto, and, for instance, the storagesystem 120 may also be configured to execute the processing forbalancing the load of the installed fibre channel switch 121 accordingto the second load balancing processing sequence RT16 described abovewith reference to FIG. 36 or the third load balancing processingsequence RT17 described above with reference to FIG. 39.

The present invention may be broadly employed in various storage systemshaving an apparatus equipped with a first function for copying data ofone logical device to another logical device paired among the logicaldevices set in itself, and a second function of virtualizing theexternal logical device and providing this to the host system.

1. A load balancing system for balancing loads of storage apparatusestherein, comprising: a first apparatus equipped with at least oneinternal logical device and configured to provide a plurality of virtuallogical devices accessible for a host system, said virtual logicaldevices being physically located on an external storage apparatus whilevirtualized as internal logical devices of the first apparatus to thehost system 1, data of at least one virtual logical device beingmaintained as a mirror-copy pair with another virtual logical deviceamong the virtual logical devices therein; a second apparatus equippedwith at least one internal logical device and configured to selectivelyprovide a plurality of virtual logical devices accessible for a hostsystem; and a management device that periodically detects a load of saidfirst apparatus and a load of said second apparatus, respectively, andmigrates a virtual logical device that is not maintained as amirror-copy pair with any other virtual logical devices in said firstapparatus to said second apparatus when the load of said first apparatusis greater than the load of said second apparatus.
 2. The load balancingsystem according to claim 1, wherein said management device, eitherafter migrating said virtual logical device that is not maintained as amirror-copy pair with said other virtual logical device therein or afterdetecting there is no virtual logical device that is not maintained as amirror-copy pair with said other virtual logical device therein,migrates per mirror-copy pair said at least one virtual logical devicethat is paired together with said another virtual logical device in saidfirst apparatus to said second apparatus to balance the loads.
 3. Theload balancing system according to claim 2, wherein, upon migrating saidmirror-copy pair in said first apparatus to said second apparatus, saidmanagement device detects current status of said mirror-copy pair andnotifies said second apparatus of control information according to thedetection result; and wherein said second apparatus sets the status ofsaid mirror-copy pair migrated from said first apparatus to said secondapparatus based on said control information notified from saidmanagement device.
 4. The load balancing system according to claim 2,wherein the migration of a mirror-copy pair of virtual logical devicesincludes transmitting pair status differential bit information from saidfirst apparatus to said second apparatus.
 5. The load balancing systemaccording to claim 2, wherein the migration of a mirror-copy pair ofvirtual logical devices includes deleting the mirror-copy pair ofvirtual logical devices from the first apparatus.
 6. The load balancingsystem according to claim 1, wherein the migration of a virtual logicaldevice includes deleting the virtual logical device from the firstapparatus by erasing an entry corresponding to the virtual logicaldevice in a management table in the first apparatus.
 7. A load balancingmethod for balancing loads of storage apparatuses, comprising: providinga first apparatus equipped with at least one internal logical device andconfigured to provide a plurality of virtual logical devices accessiblefor a host system, said virtual logical devices being physically locatedon an external storage apparatus while virtualized as internal logicaldevices of the first apparatus to the host system 1, data of at leastone virtual logical device being maintained as a mirror-copy pair withanother virtual logical device among the virtual logical devicestherein; providing a second apparatus equipped with at least oneinternal logical device and configured to selectively provide aplurality of virtual logical devices accessible for a host system;periodically detecting a load of said first apparatus, and a load ofsaid second apparatus, respectively; and migrating a virtual logicaldevice that is not maintained as a mirror-copy pair with any othervirtual logical devices in said first apparatus to said second apparatuswhen the load of said first apparatus is greater than the load of saidsecond apparatus.
 8. The load balancing method according to claim 7,wherein the migrating step involves, either after migrating said virtuallogical device that is not maintained as a mirror-copy pair with saidother virtual logical device therein or after detecting there is novirtual logical device that is not maintained as a mirror-copy pair withsaid other virtual logical device therein migrating per mirror-copy pairsaid at least one virtual logical device that is paired together withsaid another virtual logical device in said first apparatus to saidsecond apparatus to balance the loads.
 9. The load balancing methodaccording to claim 8, wherein the migrating step involves detectingcurrent status of said mirror-copy pair and notifying said secondapparatus the control information according to the detection result; andwherein said second apparatus sets the status of said mirror-copy pairmigrated from said first apparatus to said second apparatus based onsaid control information notified from said management device.
 10. Theload balancing method according to claim 8, wherein the migrating stepinvolves transmitting pair status differential bit information from saidfirst apparatus to said second apparatus.
 11. The load balancing methodaccording to claim 8, wherein the migrating step involves deleting themirror-copy pair of virtual logical devices from the first apparatus.12. The load balancing method according to claim 7, wherein themigrating step involves deleting the virtual logical device from thefirst apparatus by erasing an entry corresponding to the virtual logicaldevice in a management table in the first apparatus.